feat(ocsf): carry structured events on sandbox log stream - #3104

Review findings

• [P1] Preserve the fallback message for mixed-version gateways. For OCSF events, LogPushLayer now sets message to an empty string and carries the content only in the new ocsf_json protobuf field (log_push.rs:54-91). A pre-feat(ocsf): carry structured events on sandbox log stream #3104 gateway ignores unknown field 8 and publishes the empty fallback, so OCSF entries appear blank when a newer supervisor reconnects to an older gateway during a rollout or rollback. Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic. Please keep message populated with the shorthand representation and treat ocsf_json as additive metadata, or negotiate support before clearing the fallback.

• [P2] Expose the structured payload through the curated Go SDK. The protobuf gains ocsf_json, but types.LogLine has no corresponding field (types/log.go:9-21) and LogLineFromProto does not copy GetOcsfJson() (converter/log.go:18-25). Exporters using the curated API therefore still cannot retain the complete event. Please add a deep-copied byte field or parsed OCSF representation and converter coverage.

@johntmyers

Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic.

Is this really the case? I know previously the supervisor was always using the latest image, but do we make any guarantees about using a newer supervisor with an older gateway?

@johntmyers

Expose the structured payload through the curated Go SDK.

I think this should be done as a separate PR. Thoughts?

@johntmyers

Supervisor images can be pinned independently, so this mixed-version pairing is supported and realistic.

Is this really the case? I know previously the supervisor was always using the latest image, but do we make any guarantees about using a newer supervisor with an older gateway?

I don't think we explicitly prevent this case. Maybe for pre-0.1.0 this doesn't matter. But I think the review agent was orienting around HA rolling upgrades where the mismatch could happen.

The ocsf_json field creates a second consumption path for structured OCSF events at the gateway. In #2717 I add a configurable schema version downgrade that strips fields/profiles for SIEMs targeting older OCSF versions (1.1, 1.3). Currently the downgrade runs at the sandbox JSONL layer. With ocsf_json now traveling to the gateway, should the bytes be pre-downgraded, or should the gateway apply its own downgrade pass? If a future exporter consumes structured events at the gateway, it would need the downgraded version for SIEM compatibility.

@zanetworker My current thinking is that doing the downgrade at the sandbox is not sufficient, or scalable, because we would need to support many potential downgrades, and there can be multiple destinations for OCSF events, including OTLP, a rotated log file, and others. Throwing away information at the sandbox to satisfy a single SIEM schema is suboptimal, I think.

Someone at NVIDIA built a pretty comprehensive solution that correlates information from gateways and sandboxes, including OCSF events but also spans and other logs. This solution makes me think we should keep the gateway and sandbox implementation as simple as possible and rely on other tooling (like that solution) to do transformation of the data for various consumers. The architecture of that solution today looks like this:

flowchart LR
    A[OpenShell files] --> E[Exporter]
    B[OpenShell read-only gRPC] --> E
    C[Gateway and Relay OTLP] --> E
    D[Kubernetes context] --> E
    E --> P[Identify, validate, redact, correlate]
    P --> Q1[Persistent CloudEvents queue]
    P --> Q2[Persistent OTLP queues]
    P --> R[Redacted recovery archive]
    Q1 --> S[SIEM, security, data lake, policy system]
    Q2 --> O[Observability backend]

I spiked having different kinds of exporters built into the gateway and I feel it expands the scope of the gateway too much, particularly given how critical OCSF events are. Instead, I think we should keep it simple and have the gateway create a durable record of OCSF events using the latest supported schema version and use something like the above tooling to read that stream, do any other relevant correlation, as well as necessary PII redaction, and forward it on to external consumers. That means all the queuing and retrying behavior that we'd want would exist in the other tool, not within the gateway.

What do you think?

@krishicks Thanks for the detailed response and the architecture diagram.

Stepping back, I see the merits of the exporter approach highlighted, the concern I have is about the cumulative UX cost. Between external OCSF exporters, external policy interceptors, and other components kept outside the gateway, the operator surface area grows with each piece that's externalized. The compliance investigator persona in #1055 calls for exporting "in a schema version compatible with the organization's security tooling, without scraping human-oriented log text." Today, that persona would need to deploy the experimental OpenShell Event Exporter (which is an out-of-band tooling, probably worth understanding if there are plans of having that part of openshell) or build custom tooling to bridge that gap. Each additional component they need to deploy, configure, and maintain is friction that works against adoption.

Not trying to prescribe one way or the other, I think we are early and can be flixble/learn at the same time, but whatever the answer is, the user shouldn't need to assemble a pipeline to get schema-compatible events into their SIEM and IMO we should reduce hoops for getting there. So the question is without having it in the gateway (also how we are approaching otel relay), how do we build easy and consistent UX is the concern.

The downgrade_event() code in #2717 is a pure (JSON, target_version) → JSON transform that works wherever it runs. Happy to reposition it based on where this lands.

@zanetworker I think we'll need to get an RFC going for this because we need to get the architecture right. @delgadof is interested in proposing different approaches that will keep the gateway's duties simplified and focused and allow for it to scale, both in amount of data processed as well as with different kinds of data processing, correlation, and export.